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ABSTRACT 



Two of the most widely used academic self-efficacy 
assessment techniques, problem-referenced measurement and the Motivated 
Strategies for Learning Questionnaire (MSLQ) were compared. Participants were 
383 high school students from 4 Los Angeles (California) schools. Multi-trait 
multi -method analyses revealed that the two techniques were not measuring 
exactly the same thing. In particular, students' responses became more 
uniform in each school subject as the assessment procedures referred to more 
global events than specific problems. The two techniques also differed in 
generality. The relationship between students' verbal and mathematics 
self-efficacy perceptions was noticeably stronger with the problem-referenced 
technique than with the MSLQ. (Contains one table, three figures, and six 
references.) (Author/SLD) 



******************************************************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******************************************************************************** 



o 

ERIC 



TM028941 ED 422 379 



Effects of Scale Differences on the Generality of Academic Self-Efficacy 

Judgments 



Mimi Bong 

Ewha Womans University 
Seoul, Korea 



U.S. DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 

Z TIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 




Abstract 

Two of the most widely used academic self-efficacy assessment techniques, problem- 
referenced measurement and the Motivated Strategies for Learning Questionnaire 
(MSLQ), were compared. Multi-trait multi-method analyses revealed that the two 
techniques were not measuring exactly the same thing. In particular, students’ 
responses became more uniform in each school subject as the assessment procedures 
referred to more global events than specific problems. The two techniques also 
differed in the generality. In particular, the relationship between students’ verbal and 
math efficacy perceptions were noticeably stronger with problem-referenced 
assessment technique than with the MSLQ. 

Poster session presented at the annual meeting of the American Educational Research 
Association, San Diego, CA, April 1998. 
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Academic self-efficacy is a predictive construct that corresponds directly to the 
outcome of interest (Zimmerman, 1996). Self-efficacy researchers thus assess 
students’ efficacy beliefs toward a set of specific and particularized tasks and relate 
them to diverse indexes of achievement behaviors on the very tasks. It is noteworthy 
that most academic self-efficacy investigations so far concerned the strength of 
students’ efficacy perception and its impact on subsequent performance. Although 
absolutely important in its own right, assessing strength of efficacy beliefs toward a 
limited range of very specific tasks does not permit researchers to make inference on 
the relation between various task-specific efficacy beliefs. 

In fact, Bandura (1986) claimed that persons’ efficacy beliefs can differ along 
the dimensions of strength, level, and generality. A recent investigation on the 
generality of academic self-efficacy judgments revealed that students’ efficacy 
perceptions indeed generalize beyond the boundary of a single, specific task. Using 
confirmatory factor analyses. Bong (1997) reported that high school students’ 
judgments of their own academic competence were more or less equivalent within the 
boundary of each school subject. Evidence also showed that students’ efficacy beliefs 
might generalize even further to provide bases for two higher-order factors, verbal and 
math academic self-efficacy. Such results can provide valuable information for both 
researchers and practitioners in education, because they shed light on the possibility 
that efforts invested in educational intervention programs, especially those designed 
for bolstering students’ academic confidence in specific tasks, may bring added 
benefits to participants. 

Currently, there are two types of scales or measurement techniques most 
frequently utilized in academic self-efficacy research. One is to ask students to rate 
their confidence for successfully solving a set of specific problems presented. The 
other is to ask students to report their confidence for successful performance by 
presenting simply verbal descriptions of the tasks or domains of interest (see Pajares, 
1996). Such difference in the measurement technique and specificity has been found 
to wield notable impact on efficacy beliefs’ potency in explaining students’ academic 
performance. The impact of scale differences on the generality of efficacy beliefs has 
yet to be investigated empirically. The purpose of the present investigation, therefore, 
is to see whether academic self-efficacy beliefs in the same domains assessed by 
different techniques demonstrate comparable degrees of generality and, more 
important, represent the same construct. 

Results from the present investigation are also expected to help tease out the 
sources of difference between academic self-concept and self-efficacy research. 
Differences in theoretical stipulation set aside, there have been conspicuous 
differences in the assessment procedure typically associated with the two bodies of 
research. Unlike academic self-efficacy research that often resorts to obtaining 
students’ confidence ratings by presenting specific tasks or problems, academic self- 
concept research has relied on more general, survey-type measurement procedures. 
Several researchers noted that such difference might have been at least partially 
responsible for the observed differences in the two constructs’ relations to 
achievement as well as relations among domain-specific self-perceptions, particularly 
between the verbal and math selves (e.g., Marsh, Walker, & Debus, 1991). 

Method and Procedures 

Participants were three-hundred and eighty-three high school students (49% 
male, 51% female) from four Los Angeles-based high schools. The sample came from 
a larger study that involved 588 students (see Bong, 1997). Students (a) reported their 
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confidence for solving seven representative problems in six school subjects (i.e., 
English, Spanish, American history, algebra, geometry, and chemistry), (b) responded 
on the seif -efficacy scale of the Motivated Strategies for Learning Questionnaire 
(MSLQ), and (c) reported course grades in the six subjects. Zimmerman and others 
(Marsh, Walker, & Debus, 1991; Zimmerman, 1995) discussed that judgment of 
academic self-efficacy puts heavier emphasis on mastery criteria (i.e., being able to 
succeed) rather than normative ones (i.e., being better than others). Accordingly, the 
original questions in the MSLQ were modified so that items or statements prompting 
normative judgment of ability were excluded. The final scale contained six questions 
for each school subject. 

Results and Discussion 
Multi-Trait Multi-Method Analysis 

The present investigation involved two measurement techniques assessing 
academic self-efficacy judgments in the six school subjects. Before examining 
separately the difference due to scales on the generality of self-efficacy perceptions, a 
multi-trait multi-method (MTMM) design was first imposed on the data. The MTMM 
design allows examining (a) convergent validity of academic self-efficacy beliefs 
assessed by the two methods and (b) discriminant validity of the six subject-specific 
academic self-efficacy from each other. In particular, a MTMM design with 
confirmatory factor analysis (CFA) approach affords partitioning of the variance to the 
factor and method effects (Pedhazur & Schmelkin, 1991). Four measured variables 
were created for each of the six subject-specific academic self-efficacy factor by 
combining students’ responses on two to three problems or items. Two of the four 
measured variables for each self-efficacy factor shared the same method, providing 
twelve measured variables for each method factor. 

Figure 1 presents results of the MTMM analysis. The two method factor 
accounted for a substantial amount of the variance in each of the twenty-four 
measured variables. Loadings for the Problem method factor ranged from .40 to .97, 
whereas those for the MSLQ method factor ranged between . 1 1 and .72. The Problem 
factor, in particular, wielded equal or greater effects on the indicators as the self- 
efficacy factors, with an exception of Spanish. Students’ problem-referenced self- 
efficacy ratings were thus evenly influenced by their perceived capability in each 
school subject as well as by the nature of the problems presented. An opposite pattern 
was observed with the MSLQ method. Effects of the MSLQ scale were considerably 
less than effects from each subject-specific self-efficacy factor, again with an 
exception of Spanish. In other words, students’ responses on the MSLQ were not 
influenced as much by the nature of items. Rather, students provided more uniform 
responses in each school subject regardless of the questions asked which resulted in 
most of the variance in the measured variables being determined by subject-specific 
self-efficacy effects. 

Given the results presented up to this point, one would conclude that the 
MSLQ self-efficacy scale is more effective in assessing subject-specific academic 
self-efficacy than the problem-referenced scale. Such a claim can be qualified only 
when the two scales indeed tap the same psychological constructs. Unfortunately, 
factor loadings showed that the self-efficacy factors assessed by the two methods 
seriously lack convergent validity. Magnitude of paths emanating from each self- 
efficacy factor to its respective indicators differ substantially by the method of 
assessment used. The correlation coefficient between the Problem and MSLQ method 
factors was mere .09 (p > .05). 
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It is possible to examine the relative contribution of each construct or method 
in explaining the variance of measured variables by imposing equality constraints on 
certain parameters (Pedhazur & Schmelkin, 1991). Fixing the effects of each self- 
efficacy factor on its respective indicators to be equal to each other permits 
examination of the relative contribution of each method factor. Similarly, fixing the 
effects of a single method factor on multiple indicators to be equal to each other 
allows comparison of effects from each of the academic self-efficacy factors. Figure 2 
presents results of the MTMM analysis when the equality constraints were imposed on 
the trait effects, whereas Figure 3 presents results with equality constraints being 
imposed on the method effects. 

Both Figures 2 and 3 clearly demonstrate that the problem-referenced and 
MSLQ self-efficacy scales were not measuring exactly the same things. Figure 2 
shows that measured variables based on specific problems were more heavily 
influenced by the method effects compared to those based on the MSLQ, when the 
effects from each self-efficacy factor were set to be equal for all its measured variables. 
More variance is attributable to method effects in the problem-based than MSLQ 
indicators. The difference is more pronounced in English and American history than 
in quantitative subjects. Also, the equal-effects-from-constructs-to-indicators 
constraints left most of the variance of the problem-based indicators unexplained in 
verbal subjects. Factor loadings after standardization ranged between .04 to .07 for the 
problem-referenced self-efficacy ratings. In contrast, the MSLQ variables all 
demonstrated loadings well above .70. The hypothesized self-efficacy constructs were 
thus defined mostly by the MSLQ self-efficacy scores, when the equality constraints 
were imposed on the construct effects. 

Results of the MTMM analysis with the equal-method-effects constraints 
corroborated that the two methods were measuring slightly different constructs. When 
the effects of each method on their respective measured variables were constrained to 
be equal, each subject-specific academic self-efficacy factors shared substantially 
more variance with the MSLQ than problem-based indicators. As in all previous 
analyses, the difference in construct effects on measured variables were more 
noticeable in verbal subjects than in quantitative subjects with an exception of 
Spanish. 

Next, correlations among self-efficacy factors were examined as a way of 
exploring the discriminant validity of subject-specific academic self-efficacy from 
each other. Table 1 presents factor correlations among the six self-efficacy factors 
when the two methods were combined. As can be seen, each subject-specific 
academic self-efficacy factors correlated from -.08 to .81 to each other, displaying that 
students form more or less independent perceptions of their academic capability for 
each school subject. 

Comparison of the Generality of Academic Self-Efficacy Judgments by Each Scale 

Confirmatory factor analyses were conducted separately on each data set based 
on each measurement technique. First, six first-order models with different degrees of 
generality were fitted to the data and the goodness-of-fit indexes were compared. The 
following are the model descriptions: Model 1 with three factors (i.e., Verbal-English, 
Verbal-Spanish, and Math); Model 2 with four factors (i.e., English, Spanish, History, 
and Math); Model 3 with four factors (i.e., Verbal-English, Verbal-Spanish, Math, and 
Chemistry); Model 4 with five factors (i.e., English, Spanish, History, Math, and 
Chemistry); Model 5 with five factors (i.e., Verbal-English, Verbal-Spanish, Algebra, 
Geometry, and Chemistry); and Model 6 with six factors (i.e., English, Spanish, 
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History, Algebra, Geometry, and Chemistry). 

All six models demonstrated a better fit when they were imposed on the 
problem-based self-efficacy. The difference in fit was especially pronounced in Model 
4 that prescribed a single factor for students’ algebra and geometry efficacy ratings. 
The fit of Model 4 with the problem-based self-efficacy, x 2 (125, N = 383) = 424.67, p 
< .001 (NNFI = .94, CFI = .95), was better than the fit with the MSLQ self-efficacy, 

X 2 (125, N = 383) = 1019.85, p < .001 (NNFI = . 86 , CFI = .89). In other words, 
students made more fine distinction between their capability in algebra and that in 
geometry when asked more global questions than when provided with specific 
problems. 

Two second-order factor structures were imposed on the two data sets to 
further test the generality of academic self-efficacy judgments. There was no 
noticeable difference in fit between the two data sets when both the model with 
General Academic Self-Efficacy and the model with Verbal and Math Academic Self- 
Efficacy based on the six subject-specific factors were imposed. Overall, the superior 
fit of all first-order models with the problem-based self-efficacy judgments than with 
the MSLQ efficacy scores along with results from the second-order models provided 
evidence that students may engage in slightly different processes or assign different 
weights to diverse information sources when arriving at their competence perception, 
depending on what or how the questions were asked. 

Interestingly, the relation between Verbal and Math Academic Self-Efficacy 
also varied depending on the empirical data set utilized. Although the second-order 
model with Verbal and Math Academic Self-Efficacy accounting for the covariances 
in the six subject-specific efficacy factors exhibited comparable fit between the 
problem- and MSLQ-based data, the correlation between Verbal and Math Academic 
Self-Efficacy was substantially larger (. 66 ) with the problem efficacy than with the 
MSLQ (.43). As discussed earlier, this result has implications for the purported 
difference between academic self-efficacy and self-concept research. The MSLQ scale 
employed in the current study is composed of six items that inquire students’ 
perceived competence in each school subject (e.g., I’m certain that I can understand 
what is taught in [a specific school subject] class; I am sure that I can do an excellent 
job on the problems and tasks assigned for [a specific school subject] class). Even 
with the comparative statements removed from items, the correlation between MSLQ- 
based Verbal and Math Academic Self-Efficacy became conspicuously smaller 
compared to the one between problem-referenced efficacy factors. When we consider 
that academic self-concept research typically relies on more general measures 
compared to academic self-efficacy research, results from the present investigation 
clearly demonstrated the possibility that the specificity (or generality) of assessment 
procedures can make different facets of the self-referent thoughts more salient. 
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Table 1 

Correlations Among Academic Self-Efficacy Factors From Multi-Trait Multi-Method 
Analysis 



Factors 


1 


2 


J 


4 


5 6 


1 . English Academic Self-Efficacy 


1.00 










2. Spanish Academic Self-Efficacy 


-.08 


1.00 








3. History Academic Self-Efficacy 


.62 


-.01 


1.00 






4. Algebra Academic Self-Efficacy 


.22 


.07 


.34' 


1.00 




5. Geometry Academic Self-Efficacy 


.18 


.09 


.30 


.81 


1.00 


6. Chemistry Academic Self-Efficacy 


.22 


.11 


.37 


.61 


.73 1.00 



Note. Correlation coefficients greater than .10 are significant at p < .05. 
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Figure 1. Multi-trait multi-method model with six correlated traits and two correlated 
methods. Dotted lines denote nonsignificant paths (p > .05). Eng = English; Span = 
Spanish; Hist = American history; Alg = algebra; Geo = geometry; Chem = chemistry; 
Prob = problem-referenced; MSLQ = Motivated Strategies for Learning Questionnaire. 
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Figure 2. Multi-trait multi-method model with equality constraints imposed on trait 
effects. Dotted lines denote nonsignificant paths (g > .05). 
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Figure 3. Multi-trait multi-method model with equality constraints imposed on 
method effects. 
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